Parallel file systems are widely used in many computing environments. Parallel computing techniques are used in many industries and applications for implementing computationally intensive models or simulations. For example, the Department of Energy uses a large number of distributed compute nodes tightly coupled into a supercomputer to model physics experiments. In the oil and gas industry, parallel computing techniques are often used for computing geological models that help predict the location of natural resources.
Parallel file systems provide high degrees of concurrency in which many distributed processes within a parallel application simultaneously access a shared file namespace for a large number of clients. Parallel file systems are comprised of data servers that store and retrieve the user data as well as metadata servers that identify the data servers where particular data is stored. The metadata servers are also responsible for processing queries about the data, such as the size, permissions, and timestamps of data.
Parallel file systems typically serve supercomputers running large parallel non-interactive programs (often referred to as batch jobs) as well as a small number of “log-in” nodes which are used by interactive users (often referred to as “compile” or “front-end” nodes). Log-in nodes often experience slow interactive response times due to pending requests from the batch jobs filling the request queue of the metadata server. The interactive user therefore may experience long latency. A need therefore exists for improved techniques for processing interactive requests from users of log-in nodes.